Skip to content

fix: Filter the directories/files hashed when using commands with patterns, docs and examples fixes #720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
flora-five wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

fix: Filter the directories/files hashed when using commands with patterns, docs and examples fixes #720

flora-five wants to merge 10 commits into from
+91 −66

Conversation

@flora-five
Copy link

@flora-five flora-five commented Dec 18, 2025

Description

This set of changes does the following:

  • when hashing source paths with commands and patterns, use the patterns to filter what files are hashed
  • uses os.path.normpath to normalize other paths used as input to the prepare step
  • fixes some source paths used in README and examples
  • updates examples to version 6.x of terraform-aws-vpc to fix warnings about deprecated attributes
  • hashes again the content of package.py
  • avoids double encoding of source_path strings, with jsencode, when passed from Terraform to python

Motivation and Context

When commands and patterns are used together, the patterns filter is applied when building the archive, but not during the prepare step. Files that are not included in the archive due to the filter are still hashed. If some of those files are generated, with their content depending on the build environment, then, even if the built archives are the same across different build environments, the calculated hashes are not the same and the module wants to create again the archive and redeploy. Making the hashing step take into account the patterns filter fixes #672.

Hashing again the content of package.py restores the logic that was necessary to disable in #66 to fix #63. This time only the content of package.py is hashed, without its path.

The change to normalize other paths used as input with os.path.normpath is for consistency with existing normalization on some input paths.

Breaking Changes

None, but all the hashes will change due to inclusion of package.py into the hashing.

How Has This Been Tested?

  • I have updated at least one of the examples/* to demonstrate and validate my change(s)
  • I have tested and validated these changes using one or more of the provided examples/* projects
  • I have executed pre-commit run -a on my pull request

@flora-five flora-five mentioned this pull request Dec 18, 2025
Open
1 task
@flora-five
Change order of notes in README to have the patterns examples follow …
f502d26 
…the definition
@flora-five
Change the paths in the README examples to match the types of applica…
fe0592f 
…tions
@flora-five
Move the node-app fixture into the examples folder, where it is expec…
632f211 
…ted to be
@flora-five
Avoid double encoding of source_path strings
b94d49f
@flora-five
Normalize other paths used as input to the plan
ebf03c4 
Previously, the following paths were normalized: the working directory
for commands and the path elements when source_path is a list.
With this change, the following paths are normalized too: plain paths
given as source_path strings, path values of npm_requirements and
pip_requirements.
@flora-five
Include again into the hash the content of package.py, ignoring its path
aa193eb 
See PR terraform-aws-modules#66

TODO: remove the code and logic for hash_extra_paths?
@flora-five
Filter the directories/files hashed when using commands with patterns
3e5a6d5
@flora-five
Update to the latest version of pre-commit-terraform
dcc7f7f
@flora-five
Update path to app source
97db8e3
@flora-five
Update to version 6.x of terraform-aws-vpc to fix warnings about depr…
784f889 
…ecated attributes with version 6.x of AWS provider
antonbabenko
antonbabenko reviewed Dec 19, 2025
View reviewed changes
Copy link
Member

@antonbabenko antonbabenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good! Thanks for tackling this issue.

Could you please explain how to test it? Compare this one to the current version of the module and explain what is expected/fixed (changed or not changed).

@flora-five
Copy link
Author

flora-five commented Dec 20, 2025

Thank you for the review and feedback!

The fix can be exemplified with this code:

module "package" {
  source = "../../"

  create_function = false
  runtime         = "python3.12"

  source_path = [
    {
      path     = "app"
      commands = [":zip"]
      patterns = ["!ignore.txt"]
    }
  ]
}

and the following steps.

With the current version of the module:

$ mkdir app; echo "# source" > app/app.py; echo "initial" > app/ignore.txt
$ terraform apply -auto-approve
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.data.aws_region.current: Reading...
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_partition.current: Reading...
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
  + create

Terraform will perform the following actions:

  # module.package.local_file.archive_plan[0] will be created
  + resource "local_file" "archive_plan" {
      + content              = (sensitive value)
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0755"
      + file_permission      = "0644"
      + filename             = "builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.plan.json"
      + id                   = (known after apply)
    }

  # module.package.null_resource.archive[0] will be created
  + resource "null_resource" "archive" {
      + id       = (known after apply)
      + triggers = {
          + "filename"  = "builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.zip"
          + "timestamp" = "1766224363280384000"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.
module.package.local_file.archive_plan[0]: Creating...
module.package.local_file.archive_plan[0]: Creation complete after 0s [id=56f0b1fa178948ab0dd6b5924650a2f7a5f0caa7]
module.package.null_resource.archive[0]: Creating...
module.package.null_resource.archive[0]: Provisioning with 'local-exec'...
module.package.null_resource.archive[0] (local-exec): local-exec: Executing: Suppressed by quiet=true
module.package.null_resource.archive[0]: Creation complete after 0s [id=8964199750125432709]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
$ terraform plan
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.local_file.archive_plan[0]: Refreshing state... [id=56f0b1fa178948ab0dd6b5924650a2f7a5f0caa7]
module.package.null_resource.archive[0]: Refreshing state... [id=8964199750125432709]
module.package.data.aws_region.current: Reading...
module.package.data.aws_partition.current: Reading...
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.
$ unzip -l builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.zip 
Archive:  builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        9  01-01-1980 00:00   app.py
---------                     -------
        9                     1 file
$ echo "changed" > app/ignore.txt
$ terraform plan
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.local_file.archive_plan[0]: Refreshing state... [id=56f0b1fa178948ab0dd6b5924650a2f7a5f0caa7]
module.package.null_resource.archive[0]: Refreshing state... [id=8964199750125432709]
module.package.data.aws_region.current: Reading...
module.package.data.aws_partition.current: Reading...
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.package.local_file.archive_plan[0] must be replaced
+/- resource "local_file" "archive_plan" {
      ~ content              = (sensitive value) # forces replacement
      ~ content_base64sha256 = "5kNo8lNSAogubg0U20HitHBcM9MbBglceOHHcEd1HjI=" -> (known after apply)
      ~ content_base64sha512 = "FHbxf3F60tFrn0ILLSvUIX76GI+Jk3FWzak0xpdBwTfqQMelUynzwoUvwG7N/Ztcbul9XMIszbXxKqPqfypRng==" -> (known after apply)
      ~ content_md5          = "4864952ec7248d906d9464be266c481f" -> (known after apply)
      ~ content_sha1         = "56f0b1fa178948ab0dd6b5924650a2f7a5f0caa7" -> (known after apply)
      ~ content_sha256       = "e64368f2535202882e6e0d14db41e2b4705c33d31b06095c78e1c77047751e32" -> (known after apply)
      ~ content_sha512       = "1476f17f717ad2d16b9f420b2d2bd4217efa188f89937156cda934c69741c137ea40c7a55329f3c2852fc06ecdfd9b5c6ee97d5cc22ccdb5f12aa3ea7f2a519e" -> (known after apply)
      ~ filename             = "builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.plan.json" -> "builds/3cc66934d8b579059f506f93c8ff63885ff7b034f444d466da06a24b8850faf2.plan.json" # forces replacement
      ~ id                   = "56f0b1fa178948ab0dd6b5924650a2f7a5f0caa7" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.package.null_resource.archive[0] must be replaced
+/- resource "null_resource" "archive" {
      ~ id       = "8964199750125432709" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "filename"  = "builds/2cf974ee3774e9a68398177febb760466484c417bb82743c568de960cdbf5506.zip" -> "builds/3cc66934d8b579059f506f93c8ff63885ff7b034f444d466da06a24b8850faf2.zip"
          ~ "timestamp" = "1766224363280384000" -> "1766224449996733000"
        }
    }

Plan: 2 to add, 0 to change, 2 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if
you run "terraform apply" now.

We can see that ignore.txt is not included in the archive, but changing its content makes the module want to recreate the archive, with a different name, due to the changed hash.

With the proposed set of changes:

$ mkdir app; echo "# source" > app/app.py; echo "initial" > app/ignore.txt
$ terraform apply -auto-approve
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.data.aws_region.current: Reading...
module.package.data.aws_partition.current: Reading...
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
  + create

Terraform will perform the following actions:

  # module.package.local_file.archive_plan[0] will be created
  + resource "local_file" "archive_plan" {
      + content              = (sensitive value)
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0755"
      + file_permission      = "0644"
      + filename             = "builds/c2b94454086393d6981987a835dbd1e60d7ef7ce3290bac4f23e7bd225366a63.plan.json"
      + id                   = (known after apply)
    }

  # module.package.null_resource.archive[0] will be created
  + resource "null_resource" "archive" {
      + id       = (known after apply)
      + triggers = {
          + "filename"  = "builds/c2b94454086393d6981987a835dbd1e60d7ef7ce3290bac4f23e7bd225366a63.zip"
          + "timestamp" = "1766224679244360000"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.
module.package.local_file.archive_plan[0]: Creating...
module.package.local_file.archive_plan[0]: Creation complete after 0s [id=a14a3e217713f8311b22691fe555c72d116f39a1]
module.package.null_resource.archive[0]: Creating...
module.package.null_resource.archive[0]: Provisioning with 'local-exec'...
module.package.null_resource.archive[0] (local-exec): local-exec: Executing: Suppressed by quiet=true
module.package.null_resource.archive[0]: Creation complete after 0s [id=6751666028988243283]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
$ terraform plan
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.local_file.archive_plan[0]: Refreshing state... [id=a14a3e217713f8311b22691fe555c72d116f39a1]
module.package.null_resource.archive[0]: Refreshing state... [id=6751666028988243283]
module.package.data.aws_partition.current: Reading...
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_region.current: Reading...
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.
$ unzip -l builds/c2b94454086393d6981987a835dbd1e60d7ef7ce3290bac4f23e7bd225366a63.zip 
Archive:  builds/c2b94454086393d6981987a835dbd1e60d7ef7ce3290bac4f23e7bd225366a63.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        9  01-01-1980 00:00   app.py
---------                     -------
        9                     1 file
$ echo "changed" > app/ignore.txt
$ terraform plan
module.package.data.external.archive_prepare[0]: Reading...
module.package.data.external.archive_prepare[0]: Read complete after 0s [id=-]
module.package.local_file.archive_plan[0]: Refreshing state... [id=a14a3e217713f8311b22691fe555c72d116f39a1]
module.package.null_resource.archive[0]: Refreshing state... [id=6751666028988243283]
module.package.data.aws_region.current: Reading...
module.package.data.aws_partition.current: Reading...
module.package.data.aws_caller_identity.current: Reading...
module.package.data.aws_region.current: Read complete after 0s [id=eu-west-1]
module.package.data.aws_partition.current: Read complete after 0s [id=aws]
module.package.data.aws_caller_identity.current: Read complete after 0s [id=...]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.

We can see that after changing the content of ignore.txt, the plan shows no changes.

I would like to ask what do you think about the hash_extra_paths code and logic? Should it be kept even if it is not used or exposed to the users of the module? On one hand it was implemented in a generic way, seeing that it can do expansion of path references:

# Expand a Terraform path.<cwd|root|module> references                                                             
hash_extra_paths = [p.format(path=tf_paths) for p in hash_extra_paths]

On the other hand, for the goal of hashing the content of package.py, without its path, the hash_internal code and logic is simpler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@antonbabenko antonbabenko antonbabenko left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

2 participants

@flora-five @antonbabenko